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ABSTRACT 

Evaluation can serve a variety of educational 
management questions, but an administrator's ability to use 
evaluation as a management tool depends not only upon his or her own 
perception of evaluation but also on the perceptions of other 
potential evaluation users, in the system. Research has uncovered a 
variety of factors influencing an evaluation's use potential. These 
factors reflect human considerations, such as people's attitudes 
toward and expectations for the evaluation; context considerations, 
such as an evaluation's requirements within a particular setting; and 
evaluation considerations, such as procedures used and means of 
communicating information. The research demonstrates that an 
evaluation's use potential, and therefore its application as a 
management tool, can be greatly enhanced if someone takes 
responsibility for organizing the evaluation to meet specified needs, 
for particular users, in light of the factors operating in the given 
setting. The research also suggests that a program-level 
administrator is in the most strategic position to assume this 
responsibility. When a program evaluation is being considered, 
therefore, the administrator needs to decide on its various 
audiences, determine their questions and information needs, and 
anticipate the uses they are likely to make of the information. 
(BW) 
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THE ADMINISTRATOR'S ROLE IN EVALUATION USE* 

Center for the Study of Evaluation 
Graduate School of Education 
University of California, Los Angeles 

James Burry 
Marvin C. Alkln 
Joan Ruskus 



INTRODUCTION 

The theme of this journal issue — evaluation as a management tool in 
education is both critical and timely. It is critical because, "as the 
shift continues from the federal to the state levels in the management of 
education programs, the states become more, not less accountable for them. 
SEAs and LEAs have become accustomed to the federal government not only 
requiring the evaluation of programs but also dictating methods of evalua- 
tion" (Council of Chief State School Officers, 1982). As federal control 
of evaluation diminishes, and as federal resources for these evaluations 
also diminish, it seems likely that state and local administrators will 
need to assume greater responsibility for their evaluations (Burry, 1984). 
Given scarce resources, they will need to think strongly about the best 
ways to commit people and money to ensure that their evaluations generate 
useful information. One primary index of that usefulness will derive from 
the extent to which evaluation becomes a tool for educational management 
and decision making. 



'The research guldlna this article was supported under a grant to the 
UCLA Center for the" Study of Evaluation from the National Institute of 
Education, U.S. Department of Education. However, the findings and 
opinions expressed here do not necessarily reflect the position or policy 
of the National Institute of Education, and no official endorsement should 
be Inferred. 
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The critical 1ty of the issue is also matched by its timeliness. T*hat 
is, to suggest increased administrator responsibility in evaluation without 
offering means to channel that responsibility would be less than satisfac- 
tory. Fortunately, recent research in evaluation use (Alkin et al , 1985 1n 
press) has enabler" us to develop a framework which helps administrators to 
take a more active, Indeed proactive role, 1n organizing evaluations to 
Increase their effectiveness as a management tool 1n educational decision 
making. 

Recent attention to the quality of education 1n our schools (Boyer, 
1983: Goodlad, 1983; National Commission on Excellence in Education, 1983). 
indicates that concern with excellence 1s pervasive. That pervasiveness 
suggests that educational decision makers will need Information which 
accurately reflects the extent to which their educational systems are 
responding to the challenge for excellence. Can evaluation become a useful 
tool 1n the manage- nt of these systems? We believe that 1t can, if 
educational administrators take the kind of actions we discuss here to 
capitalize on the uses to which evaluation can be put and to ensure that 
these uses reflect their own system-level questions and needs. 

Establishing a local focus for evaluation presents little in the way 
of technical difficulties but may run counter to present attitudes toward 
and expectations for evaluation. We will elaborate some of the relevant 
issues later 1n our discussion. Suffice 1t to say, for the present, that 
if evaluation 1s to address the demonstration of excellence (or any other 
desirable quality) then there should be open discussion of issues such as 
the definition of excellence, how it is to be judged, the best means of 



demonstrating Its attainment, and the uses of evaluation In addressing 

these issues. — 

Burstein (1984) has recently discussed some applications of evaluation 

1n school Improvement efforts. While these applications can play a 
potentially central role in the definition, judgment, and demonstration of 
excellence, that potential will not be realized until administrators come 
to recognize the ways 1n which evaluation information can be used for 
management purposes. Among these purposes are: pulse monitoring - 
treating evaluation information as educational Indicators of the extent to 
* which the educational system 1s moving in the desired direction; student. 
decision making -- using evaluation Information to make accurate decisions 
about student progress and needs; program decision making - drawing on 
evaluation Information to monitor programs and services and to modify them 
asj needed; Informing educational policy -- using evaluation Information to 
guide discussion of the status of educational systems and mechanisms for 
improvement; and long- range planning -- applying evaluation information in 
decision areas such as physical plant needs, teacher hiring and assignment, 
and resource allocation (Burstein, 1984, pp. 16-19). 

As Burstein suggests, however, such management applications, which 
address Instructional Issues, support systems, and resource allocation at 
both policy and operational levels, face certain requirements. Among these 
requirements are commitment to the uses of evaluatlo^information for 
"Informed Inquiry and educational change," and "a healthy and Informed 
understanding of the limits as well as the possibilities of Information- 
based decision making" (Burstein, 1984, p. 23). We will amplify these and 
other requirements as we proceed. 

6 
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Enabling educational administrators to organize their evaluations to 
meet the uses outlined above, then, 1s the topic of our discussion. Recent 
research (AT kin et al , 1985 in press) has uncovered factors which influence 
the extent to which evaluation 1s likely to be put to use. One finding, as 
we shall see, is that the educational administrator, him- or herself, 1s a 
critical factor 1n the use process. That 1s, the extent to .which the 
administrator actively influences the direction and course of the evalua- 
tion 1s a principal determinant of the likelihood that 1t will be put to 
use. Other research (Ruskus & Alkin, 1984) suggests the kind of admlnls-' 
trative Influence likely to promote system-wide use. Recognizing and 
promoting the uses to which evaluation can be put 1s the first step 1n 
establishing its application as a management tool. 



Once accepted as a management toot, evaluation can provide a valuable 
resource for administrators who are Interested 1n finding out how well the 
system they are responsible for 1s running, and deciding whether it could 
be improved. We have/suggested that these decision needs can involve, for 



should be planned around questions reflecting the system's context, 
operations, and expectations. It must be conducted 1n ways to ensure, 
first, that these questions are answered and, second, that the answers can 
actually be put to use in making decisions about whether the system should 
continue to run as is, 1f 1t needs to be modified, what these modifications 
might entail, and what kinds of policies and resources might be required 
either for maintenance or modification. 




example, monitoring student and program decision paking, policy setting, 
and long-range planning. To meet these kinds of needs, an evaluation 
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Although there are potential obstacles facing the administrator who 
wants an evaluation emphasizing such practical uses, there are also 
organizing principles that can be applied to overcome. these obstacles. As 
we d1§cuss the administrator's role in organizing an evaluation we will 
suggest ways to strengthen Its potential for use. For purposes^ discus- 
sion, we focus our remarks primarily at* the level of a discrete education 
program such as, for example, Chapter . I, bilingual education, mathematics, 
language arts. 

Organizing for Evaluation Use 

To h§ye a high potential for use, an evaluation needs to be carefully 
planned /organized, conduct^, and communicated to Hkely users of the 
information 1t provides. This kind of evaluation rarely happens by chance; 
someone has to take the responsibility to make 1t happen. Certainly, an 
^valuator can and should take some of the responsibility for organizing an 
evaluation for use. However, 1t has become clear that the role an adminis- 
trator (e.g., a superintendent; a Chapter I program director) takes with 
regard to the evaluation has a marked effect on Its use potential. We will 
demonstrate a framework, then, that administrators, working in cooperation 
with their evaluators, can apply to gain tactical Influence over the 
direction the evaluation takes. That Influence 1s Intended to Increase the 
evaluation's potential for use 1n program management and decision making. 
Evaluation Purpose 

The framework we propose reflects a particular evaluation perspec- 
tive. That 1s, we define evaluation as a means of providing information 
that can be used to make decisions about programs. These decisions might 
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stem from questions about whether the program could be improved, they might 
reflect matters of resource allocation and monitoring, they might stem from 

i 

questions about whether the kinds of attitudes people have about the 
program could be improved. f 

Using evaluation information to assess and perhaps influence 
participant attitude is likely to make. an important contribution to 
evaluation's potential as a management tool. Some programmatic changes 
will require modification of participants' attitudes about the program or 
its evaluation before the change can be implemented successfully. That is, 
an information-bfrsed administrative decision to make some curricular 
change, to redirect resources, to reassign staff, will require staff 
support of the proposed change. 

That support will require acceptance of the information driving the 
change, and frequently use of that Information by the people who, 1n 
addition to the administrator, will play some role in accepting and 
implementing the change. Therefore, administrator ability to successfully 
use evaluation as a management tool will require collegia! support and 
information use at various system levels. 

For an evaluation to meet its potential as a management tool in a 
particular decision area, then, the administrator needs to identify other 
potential information users whose support of the decision is necessary. 
For each user or user group, the administrator will need to determine what 
questions and concerns they have with respect to the given decision area, 
and then make sure that the evaluation applies procedures and reporting 
techniques which are appropriate to the users and their questions. 



The extent to which a program administrator takes responsibility for 
Identifying the intended users, determining their questions about the 
program, shaping the evaluation procedures for answering the questions, 
deciding what kinds of Information will be collected, and ensuring that the 
Information 1s effectively communicated can profoundly affect the degree to 
which the Information can successfully be put to use. 
Evaluation Use 

By use we mean applying evaluation information to the resolution of 
the kinds of problerrs, questions, or concerns we have alluded to above. 
To be sure, evaluation can have other, perhaps unintended consequences, but 
we do not emphasize them 1n this paper. 

There are many potential* users of evaluation Information. In a school 
setting, for example, there might be a variety of programs 1n operation, 
such as: a Chapter I program; a state-funded bilingual program; a remedial 
math or language arts program designed for students 1n need of specialized 
Instruction. 

Each of these programs might be evaluated and each coul^d have a vari- 
ety of potential evaluation users. For example, let's assume that a 
district superintendent wanted to have more productive evaluations, wanted 
to be able to use evaluation Information as a management tool 1n district 
operations. The administrator might then consider ways to organize the 
evaluation to meet his or her questions and needs and those of other 
potential users. These, users, 1n addition to the superintendent, might 
include the people responsible for program operation, for Instance, such as 
the director, other administrators, curriculum developers, Instructional 
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staff, and funding agencies. Other users might consist of parents, 
advisory councils, and community organizations with an Interest in the 
program. Since each of these groups can have professional^ personal 
interests' In the program and its evaluation, each 1s a potential user of 
the information 1t provides.* A central concern 1n organizing for 
evaluation use, therefore, is the selection of the Intended users of the 
evaluation. 

Evaluation Information can be used In i variety of ways. For example, 
let's follow the case suggested above and assume that the superintendent 1s 
concerned. about the Instructional content and methods used in a remedial 
mathematics program. As a responsible manager, the superintendent has 
questions about how students are selected for the program, the extent to 
which teachers are Implementing the program as planned, the extent to which 
building principals support teachers as they attempt to Implement the 
program, the extent to which resources earmarked for the program are 
actually used 1n the program, whether or not the program seems to be 
beneficial for the students. Jhe superintendent wants the program's 
evaluation to provide answers to these questions so that he or she. can make 
information-based decisions about maintaining the program as is, modifying 
the program, maintaining, increasing, or reducing its level of resources. 

Now the evaluation may ultimately find that the program seems to be * 
running quite well. On the other hand, it may pinpoint problems and 
suggest areas for change. Regardless of the evaluation findings, staff in 
the program are likely to have different conceptions about the program. 
Some may enjoy working in the program, think it's a good one, and would 



like to see it being continued; other staff may take the opposite point of 
view. Staff are also likely to differ in their expectations for the 
evaluation. Some may want Information to help them as they carry out their 
responsibilities 1n the program; others may think that evaluation does not 
provide the kinds of information they need; some may have no expectations 

for the evaluation. 

In short, program staff, as potential implemented of the 
superintendent's decisions, are potential evaluation Information users. 
They can differ In the extent to which they have questions about the 
program, In the kinds of questions they have, and their disposition toward 
using evaluative answers. For some staff, asking them to make changes «y 
create a problem for the superintendent; for others, asking them to 
continue current practice may create a problem. 

To help preclude these possibilities and to promote the kind of 
support we mentioned earlier, Involving staff and other potential users in 
the evaluation, finding out their questions and concerns, and determining 
the kinds of information they are likely to accept and use, are crucial. 

Factors Affecting Use 

In any setting, there are many 'actors that can have an effect on 
evaluation use and therefore on Its potential as a management tool. By 
factor, we have 1n mind any characteristic or element present in a given 
situation that can affect the extent ^to which the evaluation 1s used. 
These factors stem not only from the conduct of the evaluation, but also 
from the surrounding social, political, organizational, administrative, and 
programmatic context. Factors potentially affecting an evaluation's use, 
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f„r example, Include the k<nd of role the evaluator chooses, the Intended 
users' views about the program being evaluated, the various requirements 
for the evaluation, and Its proposed methods. 

If these factors are accepted as glvens, they can reduce or negate the 
evaluation's use potential. For instance, If an Intended group of users 
firmly believes that a program could not possibly be Improved upon, it may 
be difficult to convince them to modify their view, no matter what the 
evaluation findings might reveal. On the other hand, 1f the evaluation »s 
structured and organized around Intended users and kinds of uses, and If 
the possible effects of various factors on the use potential are planned 
for, then the evaluation's likelihood for use can be greatly Increased. 

Later we win describe the full range of factors that have been shown 
to affect an evaluation's use and discuss an organizing framework 
administrators can follow to minimize negative factor Influence and 
strengthen positive factor Influence. The organizing framework, as well as 
the associated operating terms we have discussed above, grew out of our 
research on use over the past several years (Alkln et al, 1985 in press; 
Burry, 1983). 

BACKGROUND ON EVALUATION USE 
For a good number of years, the terms use or utilization have been 
cropping up 1n the evaluation literature. Up to about the mid 1970's. 
however, discussions of use relied fairly heavily on impressionistic and 
anecdotal Information. There was a lot of talk reflecting what people 
thought use looked like, with explanations often relying on speculation 
(Rossi, 1972; Mann, 1972; Cohen & Garet, 1975). 

13 
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Around the mid 1970's the picture began to change. Then we began to 
see the results of systematic research on use, research trying to di server 
what use actually means, whether or not it occurs, and what works for 1t or 
against it (Alkin, 1975; Patton et al , 1975). 

To a great extent, the careful study of use grew out of the kinds of 
promises made for evaluation. For example, evaluation was to be an Impor- 
tant tool for decision making and for Improving policy and practice. All 
the evaluator had to do, 1t was thought, was to provide valid data.. People 
would see the light and use tbe Information provided; decision making would 
be more rational and policy and ^actlce would improve. 

By now we know this was a(najve view. Certainly, Information vali- 
d1ty, especially when that term 1s mutually agreed upon by evaluator and 
potential user, can contribute to use. But so long as evaluation and its 
use were (1) seen as the s^le responsibility of the evaluator, and (2) 
expected to produce quick ^ observable, and rational decisions 1n action, 
the promise was not met. 

One of the things explaining the seeming lack of use was that for a 
long time many people thought that Information received was necessarily put 
to use, and put to use quickly. When that did not bear up 1n practice, 1t 
was assumed that no use was taking place. 

As the research was to show, however, use was occurring, though 1n a 
form quite different from and perhaps more modest than had been |xpected 
(Alkin et al , 1974; Patton et al , 1975). We began to understand that 
evaluation processes and evaluation Information usually accumulate over 
time before they are finally put to use. And even when they are used in 
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making a decision, that decision may also have been Influenced by other 

C ••• 

kinds of ^Tnformatyon' and forces outside of the evaluation. This kind of 
use can and does take place and when~t,t does 1t can help to improve 
educational dedslqji making and practice. 

However, there is something else that helps explain lack of use. That 
1s,^for use to^aTce place, we had thought, such technical factors as the 
uallty of the^valuatlon's procedures would be Important. And that 1s 
trugr. Procedural soundness can certainly contribute to use, but so can 
otherSactors, factors that are somewhat removed from the technical realm. 

For example, one early CSE finding (Alkln, 1975} showed that the 
stance taken by the eval ua*of with respect to a program's social context 
carnaffect the evaluation's use potential. Concurrent research (Patton. et 
a), 1975) pointed up the contribution to use of the "personal factor" which 
1s*typ1fie<|g for Instance, when someone takes direct responsibility for 
* trying £p make use happea** 1 

Until recently, that "someone" was usually taken to.be the evaluator, 
the "provider" of Information. Our research, howeyer, as 1t has amplified 
the "personal factor" and\ discovered others contributing to use, demon- 
strates that t-Herole of £he potential "user" of Information, such as an 
administrator, is just as important as that of the evaluator 1n promoting 
use. In many situations the evaluator him- or herself will lack the power, 
. prestige, political sensitivity, or contextual understanding necessary to 
promote use. Outwork has shown that use will frequently require the 
influence of a program administrator who does possess these and other 
attributes. 
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CSE Research on Use 

Drawing on the early studies mentioned above (Alkln, 1975; Patton €t 
al , 1975), we conducted several empirical studies of evaluation use. Among 
these were: (1) evaluation case studies; (2) an evaluator field study; and 
(3) a user survey. These studies contributed to our synthesis of the 
knowledge on use and led to a practical handbook fcr administrators^ who 
wish to organize their program evaluations for use. 

The evaluat ion :ase studies : The case studies (Alkln, Dalllak, & 

t 

White, 1979) focused, over a period of two years, on five different pro- 
grams w'\h: required evaluations. These cases provided detailed descrlp- 
tlons of school -level program Implementation and evaluation, and how the 
evaluation process unfolded 1n each program. Our analyses uncovered the 
people who shaped the evaluation process, how 1t was used 1n each case, how 
1t fitted 1n with other school operations, and how 1t Influenced 
decisions about the program. Further, by Identifying some of the factors 
promoting these usss, we were able to develop a conceptual framework to 

guide our future study of use. . 

The evaluator field study : Drawing on the emerging framework, Dalllak 
(1980) spent i year as a participant-observer working closely with three 
school -district program evaluators 1n the belief that observation and 
analyses of evaluators — the providers of Information -- at work would 
Illuminate conditions of use. By observing these evaluators at work 
Dalllak was able to elaborate some of our previously Identified factors, 
particularly those reflecting the evaluation's organizational setting, as 
well as the kinds of tactics that evaluators adopted to Increase their 
use-enhancing effect. 
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The user survey : The user survey (Stecher, Alkln, & Flesher, 1981) 
took place over the course of a year in 22 schools 1n the district in which 
the field study had previously been conducted. Our concern here was to 
characterize the role of a particular Information user, the program admin- 
istrator, 1n terms of the nature of the decisions typically confronting 
administrators, and to uncover how and what kinds of Information come to 
shape these decisions. 

The Interviews provided a picture of the kinds of decisions -- 
programmatic and other — school administrators need to make to do their 
jobs, the ways that they use evaluation and other 1n'. jrmatlon — to 
pinpoint a need, to amplify a previous conclusion — as they form these 
decisions, and the broad strategies they adopt to stimulate others to use 
Information in their programmatic responslbll 1tes. 

Synthesis and handbook : To help synthesize the knowledge on use we 
developed an annotated review of the relevant empirical and conceptual - 
theoretical literature, drawn from educat1ona| and other settings, (Burry, 
1983), and a handbook for the adm1n1strator«*Jser who plans to build use 
into his or her program evaluation (Alkln et al, 1985 in press). All of 
our work to this point Illustrated the Importance of user-evaluator 
collaboration 1n promoting use given varfbus factor Impacts. The handbook 
therefore clusters factors Into patterns which reflect the stages of the 
use process and which can be Influenced to promote use. 
Factors Affecting Evaluation Use 

On the basis of the work described above, we Identified and classified 
the individual factors affecting evaluation use into three related cate- 
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goHes - human, context, and evaluation procedure or methodology. How 
these factors Interact together determines the extent to which evaluation 

1s likely to be used. 

Figure 1 lists the three kinds of factors. Those 1n the human 
category reflect evaluator and user characteristics that have a strong 
influence on use. Included here are such factors as people's attitude 
toward and Interest 1n the program and Its evaluation, their backgrounds 
and organizational positions, and their professional styles. 

Context factors include the kinds of requirements and fiscal con- 
straints the evaluation faces, and the relationships between the program 
being evaluated and other segments of Its larger organization and 
surrounding community. 

The evaluation factors refer to the actual conduct of the evaluation, 
and Include how the evaluator and users work together, the procedures used 
1n the evaluation, and the quality of the Information 1t provides. 

The factors 1n each of the three groups have a demonstrated Importance 
to use, and many of them will require administrative Influence to promote 
use. In the next section of the article, therefore, we offer a series of 
observations drawn from the empirical studies of use. These observations 
help define each of the factors 1n Figure 1 and suggest the kinds of 
Influence they may have, as a precursor to discussion of factor Interaction 
patterns and administrative organizing to promote use as a management 
tool . 

Observations Drawn from Empirical Studies 

With respect to the human factors affecting use, an evaluation's use 
potential 1s likely to Increase to the extent that: 

18 
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Figure 1; Factors Affecting Use 
I* Human Factors 
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A. £ valuator Characteristics 

1. commitment to use 

2. willingness to Involve users 

3. choice of role 

4. rapport with users 

5. political sensitivity 
• 6. credibility 

7. background and identity t 

a. gender 

b. title 

8. User Characteristics 

1. identity 

a. range of potential users 

b. organizational positions' 

c. professional experience levels 

2. interest in the evaluation . 

a. views about the project being evaluated 

b. expectations for the evaluation 

c. predisposition toward the evaluation 

d. perceived need 
^ e. perceived risks 

3. commitment to use 

4. professional style 

a* administrative and organizational skills 

b. Initiative . 

c. openness to new ideas or change 

5. Information processing 

a* preferences for particular for»s 

b. how Information 1s processed 

21. Context Factors 

A. Pre-existing Evaluation Bounds 

1* written requirements 

2. other contractual obligations 

3. fiscal* constraints 

B. Organizational Features 

I. mtraorganl national 

a. role of central /district office 

b« Interrelationship between unit and central /district 
administration 

c. institutional arrangements 

d. unit level autonomy 

e. sources of information beyond evaluation likely to be 1n use 
f • perceived Institutional risk 

2* external features 

a. community climate 

b. community influence 

c. role of other agencies 

C. Project Characteristics 

1. age/maturity 

2. innovetlveness 

3. overlap with other projects 
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III. Evaluation Factors 

A. Evaluation Procedures 

1. methods used 

a. appropriateness ' 

b. rigor 

2. dealing with mandated tasks 

3. used of a general model 

B. Information Dialogue 

1. amount and quality of Interaction between evaluator and users 

C. Substance of Evaluation Information 

I* Information relevance 

2. Information specificity 

0/ Evaluation Reporting 

1. frequency of Information provided 

2. timing of Information 

3. format of presentations 

a. oral presentations 

b. written reports 

c. statistical and narrative data 
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The evaluator -- 

° 1s personally committed to seeing his or her work put to use, and 
actively makes efforts to facilitate the use of Information; 

° 1s willing to involve users 1n the evaluation through cooperative 
planning and conduct of the evaluation and Its uses; 

* recognizes that alternative evaluation roles exist, chooses a role 
that 1s appropriate 1n the given setting, and focuses on serving 
program needs and questions 1n addition to any external 
requirements; 

° develops rapport with users by earning thel^truSt 1n *n atmosphere 
of harmony anq agreement; . ' ' \ J 

° 1 s politically sensitive to the program and understands the rel a- 
tlonshTp among formal and Informal power sources, opinion makers, 
decision making processes, and the function of evaluation as one of 
the Inputs to these processes; 

° establ 1 shes credl I b1 1 1 ty 1 n teems of technical competence and 
personal and professional manner. 

The users — 

° are clearly Identified so that the evaluator understands the range 
of organizational positions and professional experience levels — 
administrative vs. operational , sole or snared decision-making 
authority, familiarity with evaluation — which are represented 
among the users and which bear on their potential for using 
Information; 

° view the project 1n such ways that they would be willing to modify 
these views, if warranted ; ~^ 

0 have specific expectations for the evaluation -- determining the 
program's efficiency, understanding Its processes, assessing Its 
outcomes -- which are translated Into questions and concerns that 
the evaluation will address; 

° are predisposed to accepting the evaluation's findings, which may 
be because they \ 

° have a high perceived need for evaluative answers to their 
questions, and" 

0 perce ive the risks of the evaluation as outweighed by the potential 
benefits. In addition, they 



20 



18 

i 

» 

0 are personally comml tted to using evaluation Information as their 
questions and concerns are answered, and 

° have sufficient administrative and organizational skills to act on 
Information, to get things done. They will 

0 take the Initiative to use evaluation Information 1n their own area 
of responsibility and, 1f necessary, to stimulate others to follow 
their example. Further, they 

° are open to new Ideas or change that stem from the findings, even 
1f these findings suggest they need to modify their original views 
of the project. And, as the evaluation process unfolds, their 
positive Interest 1n the evaluation remains high, because they 

0 ask for and receive the kinds of Information they prefer to use -- 
narrative, descriptive, or some combination, through the kinds of 
processes — oral reports, written reports, detailed or summary 
treatments, they are most comfortable or routinely familiar with. 

With respect to the context factors affecting use, an evaluation's 

use potential 1s likely to Increase to the extent that: 

1. The pre-ex1 sting evaluation bounds — 

0 are characterized by a guided harmony rather than by conflict and 
tension. The evaluation's wri tten requirements -- legal codes, 
federal /state requirements — permit sufficient flexibility so that 
the evaluator can respond to such other contractual requirements as 
those set by program administrators or operators. ^ — - 

2. The organizational features — - 

0 are marked by amicable co-existence 1n an atmosphere stressing 
discussion and the negotiation of problems and needs; 

0 facilitate the central /district office — often the evaluation 
sponsoK — role in balancing broad system concerns with those of 
the Individual units, such as the schools who. are subject to 
evaluation; ^ 

0 permit sufficient unit level autonomy so that unit (e.g., a school) 
questions receive a fair share of the evaluator 1 s attention as he 
or she addresses a variety of broad organizational and unit 
questions of Interest; 

° promote frank discussion of the perceived institutional risks and, 
where there 1s a question of whether the evaluation benefits will 
outweigh the risks, consider the possible outcomes and resultant 
actions the organization might take; 




0 are free from undue or negative Influence from the surrounding 
community or other agencies . 

3. Program characteristics — 

° are clearly defined on such dimensions as age/maturity , 

innovativeness , and overlap with other programs because these 
characteristics have a bearing on the kinds of procedures the 
evaluator should select and the kinds of information he or she 
should provide 1n order to stimulate use. 

With respect to the evaluation factors affecting use, the use 

potential 1s likely to Increase to the extent that: 

1. The evaluation procedures — 

° are appropriate to the particular project. A selected procedure 
must be appropriate as a method for addressing the given question, 
and also appropriate 1n the context of the project; 

* address the matter of rigor from the dual standpoint of accepted 
standards of evaluation practice and the users' conception of what 
constitutes rigor; 

• deal with mandated tasls -- funding agency requirements, central 
office needs, unit level questions — , 1n a balanced manner so that 
no single point of view 1s seen to dominate; 

° reflect the viewpoint that no single evaluation model 1s Inherently 
superior; Instead, evaluation 1s seen as a tool for decision making 
and the selection of evaluation procedures 1s guided by the 
decision-making process. 

2. Information dialogue — 

° refl ects purposeful , guided sharing of ideas between eval uator and 
users; 

0 1s ongoing, 1n sufficient amounts to stimulate or maintain user 
Interest In the eval uat1 onTwTft Tqual 1 ty growing out of 
coll eg1 all ty and reciprocity. 

3. Evaluation substance -- 

° 1s relevant from the users* standpoint because 1t constitutes 
pertinent answers to the questions they have raised; and 

0 is specific by focusing Its content on the needs and interests of 
the particular user or user group; 
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4. Evaluation reporting — 

0 is marked -by frequent and well-focused provision of Information^ 

° Is timely in that 1t reflects program chronology and meshes with 
Important events stemming from the program's decision needs; 

0 uses whatever variety of presentation formats -« *. oril, written, 
statistical /narrative, formal or inrormai that is appropriate to 
the range of users and their evaluation interests. 

J 

Factor Interactions 

... f* 

The preceding observations, begin t6 suggest that factors are likely to 
Interact to affect use. Here we win discuss a few posstble Interaction 

•. •••-«' . ■ 

\ • / 

patterns to Illustrate the kinds of phenomena the administrator might need 
to consider as he or she organizes the evaluajHon for use, primarily 
because many of the factors are beyond the evaluator '^s control. 

For example, to help promote program-level use, the evaluator should 
address questions relevant to the program, questions of Interest to program 
staff. The extent to which the evaluator 1s successful will depend, 1n 
part, on the various requirements for the evaluation, such as those set by 
a funding agency, and whether any particular requirement is allowed to 
dominate. But it will also depend on users' Interest in the evaluation and 
their commitment to applying Its findings. However, users' predisposition 
to make this application can be affected by perceived institutional risk, 
pressures from the program's community, and the timing at which reports are 
provided, to mention but a few of the possibilities. 

Many of the factors and Interactions suggested above may not be 
amenable to evaluator Influence. For example, while the evaluator may 
commit him- or herself to use, the associated user commitment, which also 
contributes to the application of results, 1s properly 1n the admini- 
strator's sphere of influence. 

2J 
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In short, to the extent that the factors mentioned above are subject 
to Influence 1n a given setting, many are jn the administrator's domain and 
are therefore perhaps more amenable to his or her Influence. And this 
influence, 1f necessary, can cut across all three factor categories, not 
only the context/organizational category traditionally associated with 
administrative responsibility. 

We suggested earlier that to be able to use evaluation as a management 
tool, as a decision-making tool, 1t 1s critical that those people who may 
be affected by a particular delslon be Involved In the decision-making 
process. What do the kinds of interactions noted above, then, suggest for 
the admlnlstrator-organ^er trying to Increase an evaluation's potential as 

a management tool? ' 

First of all, by very virtue of his or her entry Into the use process, 
the administrator becomes one of the factors Influencing use. Continuing 
CSE research on factors promoting high evaluation utilization has suggested 
kinds of evaluator behavior which promote use. These behaviors offer clues 
to the kind of overall demeanor that the administrator-organizer might 
adopt, first of all , to create an atmosphere conducive to evaluation use. 

We recently analyzed several evaluations whose high utilization levels 
were documented as part of an AERA award to recognize such evaluations 
(Ruskus a Alkin, 1984). Many of the factors cited tended to confirm those 
displayed 1n Figure 1. Five of these factors, each of which was cited as a 
use-promoting characteristic, suggest how professional style can have a 
bearing on use. These five factors are level of effort, leadership 
behavior, user Involvement 1n the evaluation, Involvement 1n Implementing 
recommendations, and commitment to use. 

24 
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Evaluation users frequently cited the high levels of evaluator effort 
that contributed to their use of results. From the standpoint of the 
equity theory of inflation (Adams, 1965; Weick 5 , 1966) 1t seems likely that 
when level of evaluator effort is deemed to be high, users demonstrate high 
util 1zat1on. 

Leadership was another factor cited as contributing to evaluation 
tpe. Social psychologists such as Ukert (1961) and StogdUl (1974) 
suggest that leadership may be seen as originating new Ideas; mixing with 
other participants; acting on behalf of other participants; reducing 
conflicts; organizing; communicating; recognizing participants' efforts; 
stimulating participants to achieve; and helping them carry out their 
duties. ' 

Involving the potential users was another frequently cited factor 1n 
the highly utilized evaluations studied. Beyond the jidea that users are 
likely to use Information when they play a part 1n generating the 
Information, participant management theory (Ukert, 1967) suggests that 
supportive relationships, group declsl on-making, and shared organizational 
objectives contribute to commitment to carrying out organizational policy 
and decisions. 

Evaluator Involvement 1n implementing recommendations also played a 
role 1n the highly utilized evaluations. Such behavior, taking place after 
the report was generated, can rui^the gamut from interpreting implications 
of a recommendation to making concrete suggestions about areas 1n need of 
improvement and possible means of promoting such Improvement. In this 
vein, sociological theory on the management of change (e.g., Keen & Scott 
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Morton, 1978) suggests that evaluation needs to be concerned with 
Introducing the need for change, striking out 1n the direction of change, 
and Integrating the change Into existing frameworks. 

User commitment to use, finally, seemed Important in all the highly 
utilized evaluations. On the basis of themes Identified in the marketing 
literature (e.g., Rogers,' 1962), the users in the evaluations studied can 
be typified as "early adopters" who were (or became) highly disposed to try 
out new Ideas. 

Mow, 1n several Important ways, the adminlstratO^ try 1 ng 16o promote 
evaluation use 1s assuming evaluation-like responsibilities. To the extent 
that such 1s the case, then administrator efforts in promoting use, 1n 
providing leadership to other potential users and involving them in the 
evaluation and 1n Implementing its recommendations, will help to stimulate 
their commitment to use. That commitment, however, 1s likely t© be 
short-lived unless the evaluation, or at least part of the evaluation 
effort, 1s focused on user concerns. 
Establishing/ a User Focus / 

Previously we outlined some functions of evaluation that would enhance 
its relevance atfd use as a management tool. We have suggested that, to the 
extent an evaluation 1s to serve multiple audiences — funding source, pro- 
gram director and staff — then the needs of these various audiences need 
to be recognized and kept 1n proper balance. To be used as a management 
tool at the local system level at which the evaluation 1s conducted, the 
evaluation must be organized so that, 1n addition to satisfying any other 
requirements, it Identifies, addresses, and answers local -level questions. 
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We mentioned earlier that while establishing a local focus presents 
few technical difficulties, 1t may encounter some attltudinal barriers 
reflecting conflicting sets of evaluation expectations. For example, let's 
resume the case of the superintendent responding to the Issue of excellence 
1n education. What problems might he or she encounter 1n attempting to 
establish local needs as one of the fod of the evaluation? In attempting 
to get evaluation Information that has local management relevance? 

Consider the kinds of management concerns we alluded to earlier 
(Bursteln, 1984). In terms of drawing upon evaluation to provide Indi- 
cators of progress, who 1s to decide what these Indicators are to be? Are 
there conflicting viewpoints? Can they be reconciled? Can they legiti- 
mately differ 1n various settings? 

In regard to making decisions about students, what kinds of decisions 
are to be made? Are decision needs -- such as comparison versus Individual 
diagnosis -- 1n competition? Is one kind of measure deemed superior to 
another? Is that viewpoint based 1n fact or does it grow from tradition? 

With respect to program decision making, does the evaluation have, to 
address multiple audiences? Are there potential conflicts between them? 
Can the evaluation reconcile external accountability concerns and local 
concerns about program monitoring and Improvement? 

With respect to other possible management applications of evaluation 
~ long-range planning and policy formulation it 1s unlikely that evalu- 
ation can, or should, be used at the local level unless 1t first has local 
relevance on the other three Issues outlined above. Can evaluation come to 
have that local relevance? 
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While the reduction 1n federal controrof evaluation that we alluded 
to earlier 1s Intended to Increase local — SEA and LEA — responsibility, 
can we assume that the federal Intention 1s accepted "at these levels? 
Further, can we assume that SEAs and LEAs responding to the possibility of 
assuming greater control of their evaluations will be 1n agreement on basic 
Issues such as evaluation purposes, emphases, and procedures? If not, will 
one point of view dominate and thus reduce the evaluation's relevance for 
the other? What are some of the Issues confronting our school district 
superintendent who wants to make sure the evaluation will serve his or her 

management and decision concerns? 

One primary Issue, as we have already suggested, 1s that the evalua- 
tor's ability to focus on one set of needs may be constrained by other 
factors 1n the setting. It may be that one set of needs, requirements, or 
dominant attitudes causes the evaluator to adopt a certain role and collect 
certain kinds of Information which, 1n turn, may cause the superintendent, 
and his or her colleagues, to view the evaluator and the evaluator* s work 
with something less than enthusiasm. We believe that the current situation 
with respect to evaluation fod requires the superintendent's attention. 

First, some of the superintendent's potential evaluation users may not 
be convinced that changes In federal requirements will actually reduce 
external supervision and control. That 1s, while federal supervision may 
decrease, the state may continue or Initiate, or be perceived by LEAs to be 
continuing or Initiating, policies which offset LEA attempts to direct 
their evaluations toward LEA matters. 
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Second, some of the superintendent's potential evaluation users may be 
so thoroughly Immersed 1n the huslness of administering a previously 
required test, perhaps a commercially published, norm-referenced test, 
tjat they are unresponsive to the possibility of developing a more locally" 
relevant test, perhaps a criterion-referenced test of a particular content 
area. * 

Third, decreased federal control 1s accompanied by decreased federal 
funds, and with reductions 1n resources, local school districts may be 
unable to supply sufficient evaluation expertise across the various content 
areas they either need to, or would like to, evaluate. An evaluator may 
need to take responsibility for simultaneous evaluations of Chapter 1 and 2 
programs and of other programs such as bilingual education. 

Related to the above Issues 1s the *» r .<as1s on technical procedures 
that still exist 1n the regulations accompanying some programs. This 
potential problem area, though 1t 1s of particular concern 1n bilingual 
programs, 1s seen to some extent in the "sustained effect" provision 1n the 
Chapter 1 regulations. CoupOj with dwindling local resources and 
evaluation expertise thinly stretched, a school district facing multiple 
program evaluation needs of a technical nature may find 1t difficult to 
comply. 

Further, the "objective measures" mentioned 1n the Chapter 1 require- 
ments may not be uniformly understood. Owing to historical precedent, the 
evaluator of an LEA program may believe that a test must be norm- referenced 
in order to be considered objective and may continue to use this kind of 
test even when 1t serves no relevant local purpose. If this 1s the case, 

* 
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the evaluation results are likely to be seen by program staff as having 
Uttle practical value for them. 

In another LEA, also concerned with the matter of objectivity, the 
pros and cons of various kinds of tests may be discussed at length without 
ever resolving the matter to the satisfaction of all potential lifc. So 
much time may be spent on the debate over the test question that the eval- 
uator has Uttle time left for planning and conducting an evaluation around 

more Important Issues. 

In districts meeting the kinds of problems outlined above, that 1s, 
where testing Issues are difficult to resolve, where there 1s limited 
expertise to balance technical adequacy andjocal relevance, there 1s 
likely to be some negative effect on factors promoting use: anxieties and 
sense of risk may dominate; program staff may believe their questions and 
concerns are receiving insufficient attention; the evaluator's credibility 
1s likely to suffer; commitment to evaluation use and perception of useful- 
ness will decrease. 

Problems such as those outlined here do appear to warrant the atten- 
tion of the superintendent 1n the case we are using for illustration. For 
example, we found 1n a recent exploratory study (Burry, 1984) of directors 
and staff members of school district research and evaluation units a 
general agreement concerning the kinds of problem areas described above. 

For example, there was a general concern about the force of historical 
precedent. Because of earlier state and district preferences (which grew 
out of earlier federal emphases), district evaluators were still required 
to administer norm- referenced tests for reporting purposes. But informa- 
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tlon produced by such tests was of limited use to them 1n carrying out 
their responsibilities. Further, the evaluators also agreed that these 
tests did not address teachers' Interests or provide the kinds of Informa- 
tion needed to monitor and adjust classroom Instruction. Most evaluators 
felt that a good deal of their limited time and resources were given over 
to generating Information which was state required 1n response to Chapter 1 
regulations. 

A few districts were trying to distill some locally-useful Information 
from a norm- referenced test. But the evaluators felt that too much time 
was Involved 1n having the tests scored and then returned for analysis and 
Interpretation so that they would be of some use to curriculum specialists 
and classroom teachers. Consequently, these efforts detracted from the 
time they needed to generate Information more specific to Instructional 
needs. 

Several districts, 1n addition to administering and reporting the 
results of a norm-referenced test, used district-developed objectives-based 
measures and/or the tests accompanying curriculum materials. (See Burry, 
et al, 1982, for a discussion of how widespread this practice 1s, as well 
as some of Its Implications.) While districts found the Information from 
these additional tests Instructional ly useful, they asked why they had to 
conduct what was, 1n effect, a parallel evaluation. That 1s, for the 
reasons suggested above, they felt they should use a *' respectable" norm- 
referenced test for external reporting purposes; given the limitations of 
the information provided by such tests, however, they felt at the same time 
that they had to resort to other devices for locally useful Information. 

U 1 
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The question raised by the evaluators was as follows: 1f such 
district-level efforts are necessary to produce formative data for district 
purposes, cannot these efforts also be used to satisfy external (summatlve) 
requirements? They asked this question even though they were aware that 
federal requirements Impose no particular kind of test; perhaps they (or 
their superiors) continue to believe that anything other than a standar- 
dized test 1s unacceptable. 

This situation created a double dilemma. On the one hand, the 
evaluators realized that, to be locally useful, the evaluation should 
provide different types of information for different groups of users and 
that such an effort takes time: time to Identify the needs and questions 
of various potential uses, time to develop or select appropriate tests or 
design other data collection procedures, time to wfn user support for the 
evaluation. On the other hand, the felt need to run a separate evaluation 
for external purposes also takes time, time that might be better spent 
addressing local questions and needs. 

Finally, while some districts were attempting to Increase their 
evaluation's local relevance, the evaluators stated that they had a hard 
time convincing other staff, such as resource specialists and teachers^ to 
become Involved 1n the conduct of the evaluation. Because of what they 
knew or believed about previous evaluations, personnel were hesitant about 
raising their own evaluation questions, reluctant to participate 1n the 
process of devising ways to answer them, and unwilling to believe that the 
evaluator would want to help them 1n the task of carrying out their 
day-to-day responsibilities. 
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In essence, then, then evaluators we Interviewed are describing how 
some potential evaluation users, the kinds of people whose support will be 
needed 1n order for the superintendent to be able to use evaluation to help 
manage his or her district's operations, may act with respect to the 
evaluation, Its conduct, and Its uses. They suggest some of the ways the 
factors we Introduced earlier may Interact to affect evaluation use. They 
suggest some likely factor patterns that any administrator will need to 
consider as he or she begins the task of organizing for evaluation use. 

ADMINISTRATIVE ORGANIZING TO PROMOTE USE 
Figure 2, which Is excerpted from the handbook we described earlier 
(Alkln et al, 1985, In press), places the factors that we believe are 
central to use In most evaluation contexts Into a pattern which will 
facilitate organizing for use. In this pattern the factors are grouped to 
reflect stages In the process of planning for and conducting an evaluation 
to maximize Its use potential. 

The factors and their potential Influence oi use should be considered 
from the standpoint of the Intended users/uses, gathering Information that 
will help the evaluator focus on these users/uses, and gathering Informa- 
tion that will help the administrator-organizer ensure that factors 1n the 
setting do not Impede that focus. 

The administrator-organizer may want the evaluation to provide 
information that he or she, and other potential Risers, can apply to one or 
several decision concerns — broad monitoring Issues, and/or student or 
program decisions, and/or planning and policy needs. The major tasks, 
then, are to decide on the users/uses on which the evaluation should focus, 




FIGURE 2: 

Factor Pattern For Evaluation Use 



A. Setting the Stage 
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asking questions 1n light of the situational factors that might Influence 
use, and using the answers to these questions to help guide both the 
evaluation process and the administrative tactics devised to help ensure 
that the evaluation stays on target. 

The administrator who assumes the use-organizing responsibility can 
use the factor pattern 1n Figure 2 — with any appropriate emphasis, 
addition, or deletion of factors given the particular context — while he 
or she considers the program, Its evaluation, the setting 1n which 1t takes 
place, and the Intended uses. This consideration Involves anticipating the 
effects that a particular evaluation direction, once taken, 1s likely to 
achieve. It consists of asking oneself a series of questions 1n light of 
the listed factors with the Intention of determining how the program 
embodies each factor; that 1s, deciding whether that embo^ient 1s likely 

t 

to have a positive, neutral, or negative effect on the Intended uses, and 
then devising strategies to strengthen or maintain positive effects while 
minimizing negative effects. These strategies may then be Implemented by 
the administrator and/or the evaluator or some other potential user. 

For example, assume that the superintendent 1n our case Illustration 
goes through the process described above and, using the scheme suggested in 
Figure 2, asks him- or herself the following question about the first 
organizing issue -- setting the stage: "As part of the 1ntra-organ1za- 
tlonal features, is there any perceived staff risk that might hinder my 
using the evaluation to plan future Instructional offerings In response to 
changing student enrollment patterns? After due deliberation and discus- 
sion with potential staff users, the superintendent discovers that some 
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teachers feel that the evaluation poses a risk to them and that the degree 
of risk 1s likely to outweigh possible benefits. Other teachers either 
feeUthat the evaluation presents little or no risk or that benefits to be 
accrued outweigh any possible risks. 

Now, this user group's reluctance to accept and apply evaluation find- 
ings 1s one manifestation of a two-part problem. First, their acceptance 
and application of Information may be Important to a larger decision area, 
one that may need to be made consensually by all users. Given the reluc- 
tance of one segment of the decision-making group to participate, 1t may be 
that the resultant decision concern 1s never fully resolved. 

Second, those with the sense of risk may advance beyond reluctance to 
participate to outright attempts to convince others of potential dangers. 
If they are successful, then Initially receptive users may later opt to 
remove themselves from the evaluation effort and, further, may attempt to 

thwart the entire effort. 

In such a situation, the superintendent organizing for evaluation use 
would need to ask other questions in order to determine: the reason for 
the sense of risk on the part of one user group; whether or not that 
perception 1s justified; the extent to which the group in question may 
attempt to convince others of the Imminent risk; the likelihood of 
success. He or she would then need to devise appropriate strategies given 
the answers to the preceding questions. 

For example, 1t may be that the sense of risk 1s unjustified or has 
become magnified, perhaps on the basis of some previous evaluation experi- 
ence. In this situation, the superintendent would need to convince the 
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hesitant group that this perception 1s unjustified so that the evaluator's 
credibility does not suffer and' the necessary user group Involvement 1s 
achieved. 

To the extent that the superintendent him- or herself encounters 
difficulty 1n minimizing sense of risk, then 1t may be possible to enlist 
trusted and respected staff members from among the more receptive users to 
help convince their colleagues that, 1n this particular setting, the risk 
factor 1s unwarranted and that participation 1n the use process 1s justi- 
fied and Important to the larger Institution. 

Keeping the above potentially Inhibiting factor example 1n mind, and 
the kinds of question-raising process and associated strategy formulation 
the superintendent considered, we will now suggest a few possible ques- 
tions, and how they might be addressed, for factors 1n each of the four 
stages 1n the use process, as depicted 1n Figure 2. These questions are 
intended to guide administrative organizing for evaluation use, and their 
answers, as with those of all the factors displayed, should Inform the 
administrator's selection of strategies to build use Into the evaluation. 
Setting the Stage 

Setting the stage Involves determining, before the evaluation planning 
process begins, the kinds of factor Interactions likely to affect use 1n a 
given setting. While these factors may be set to some extent, they are not 
necessarily "glvens." Note 1n Figure 2 that this determination considers 
possible effects stemming from the pre-existing evaluation bounds, the 
potential users Identified, program characteristics, and 
1ntra-organ1zat1onal and external features. 
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Questions that the adm1n1s,trator-organ1?er might raise here could 

include, for example: 

° Who are the Intended users of the evalutlon Information? 

° Are the pre-existing evaluation bounds such that there may be 
potential conflict, real or perceived, between program expectations 
and other requirements? 

° How 1s the program best characterized with respect to Its maturity, 
1nnovat1veness, and overlap with other programs? 

Now, let's narrow the focus a little and add some context before we go 

any further. Suppose tha$ our district superintendent had a programmatic 

concern to resolve. In the district, enrollment 1n math classes 1n some 

high schools has been dropping off sharply 1n the last two or three years; 

1n others, math enrollment -1s staying relatively constant, even Increasing 

a little. Board and parental concern with students' technical literacy 1s 

on the rise. The superintendent would like the required evaluation of the 

math program to help explain the different enrollment patterns and discuss 

what might be done about it. 

Who might the Interested stakeholders, and hence potential evaluation 
users, be? At the least: the funding agency and the board; building 
principals, math department chairs, math teachers; parents and students; 
district office math specialists. 

What might be some possible conflicts among these users? Should the 
superintendent consider this question in light of program maturity/ 
1nnovat1veness to help Illuminate the possibility of conflicting 
expectations? Very definitely. Consider the following: 

0 The board and the funding agency expect the district to continue 
reporting the math program results In terms of student scores on 
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the norm- referenced test that has been in use 1n the district for 
the last five years; parents are used to seeing these results 
discussed 1n the local newspaper; some parents want these scores to 
go up; others are asking why their children did not have to take 
the test. 

The superintendent, after meeting with math teachers n both 
reduced- and maintained/increased enrollment schools begins to get 
the distinct impression that the district's "math ^program" does not 
look the same across all schools. In some schools, especially 
those with high enrollment levels, 1nnovat1veness seems to be -the 
defining feature. But 1nnovat1veness seems to differ 1n these 
schools. In a school or two, teachers rely heavily on tests they 
have developed themselves to make decisions about Instruction; they 
treat the norm- re fere need test as something that has little rele- 
vance for them. In some of the low-enrollment schools, teachers 
stress the importance of the norm-referenced test to their students 
and emphasize Its content 1n their instruction. A few teachers 1n 
each kind of school do not fit the general pattern. 
The superintendent would like to explore these differences 1n the 
next year's program evaluation. First, he would like to have the 
norm- referenced test requirement waived for that year. In jts 
place, he would like to conduct Intensive observational studies of 
high school classroom math practice to find out 1f different 
teacher approaches to math Instruction and/or math assessment might 
help explain different enrollment patterns. 
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° U111 the funding agency accept this plan? Will the^ school board? 
How about teachers and parents? If the norm-referenced test cannot 
be waived, could It instead be administered on some sample basis fo 
the coming year? Might the evaluation apply and analyze the 
norm- referenced test and also conduct the intensive obse-vational 
study? Will resources permit this? 

° If the evaluation 1s permitted to emphasize the observational 
component, and discovers that a certain Instructional approach 
seems to be more effective than others In attracting and maintain- 
ing student enrollment, how might the board react to this finding? 
How might teachers respond? How might it be received by the 
funding agency? 

While questions such as these might all be considered 1n the Initial 
organizing step — setting the stage — some of them may not be resolved 
until later on 1n the use process. 
Identifying/Organizing the Participants 

After setting the stage for evaluation planning has taken place, a 
series of questions which amplify user characteristics such as interest in 
the evaluation and commitment to its use, as well as questions reflecting 
relevant evaluator characteristics, should be raised. This process should 
result in the formulation of the evaluator* s role and the evaluation 
procedures, carefully matched to users* Interests, expectations, and 
professional styles, which will be used. 

Among the questions that ought to be considered at this stage are: 

0 Are the Intended users committed to use and, 1f so, is their 
commitment rhetorical or real? 
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0 What do the Intended users expect from the evaluation; are these 
expectations likely to affect their desire or ability to aDolv 
Information? * pp y 

What would be the most appropriate role for the evaluator to take 
with respect to the program, and will the evaluator be willing and 
able to assume this role? 

What kind of evaluation procedures will provide the best match with 
users' professional styles? 

Let's pause for a contextual breather again. Continuing the 
superintendent's scenario, what are some of the issues of concern 1n this 
second organizing stage? 

Let's start by thinking about the evaluator for a moment. Though a 
highly-skilled professional, 1s there anything 1n his or her personal 
comportment that would cause anxiety among teachers whose classrooms were 
being observed? If so, would the evaluator accept a carefully phrased 
suggestion about classroom entry? 

Now let's take up a possible pol 1t1co/methodolog1cal problem. Let's 
assume the observational component was sanctioned by the funding agency. 
Let's also assume that the district needs all the resources it can get to 
continue Its math offerings. Let's also assume that a well-defined 
classroom practice did seem to account for student Interest 1n math. 
Should the superintendent recommend that this particular approach 
implemented district-wide, 1s 1t likely that the agency, although they 
sanctioned the observational study, would find Its results to be credible? 
Is it possible that their understanding of observational data would affect 
their refunding decision? Would the board's reaction to the superinten- 
dent's recommendation be Influenced by their financial concerns? How mlgfttf 

X 

\ 

teachers whose practice will be affected respond to the recommended 
change? 

A -, 
l x 1 
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These kinds of questions need to be resolved before evaluation 
procedures are selected and put into operation. 
Operational 1z1ng the Interactive Process 

Up to this point, the administrator-organizer has been anticipating 
future evaluation actions and effects; 1n this third stage the carefully 
planned evaluation procedures are put Into effect. The centra! factor 1n 
this group, execution of evaluation procedures, will temper all other 
factors grouped here. 

Among the questions that should be considered -are: 

° What 1s the most effective data-collection schedule, and are there 
any possible Impediments to this schedule? 

° Do any of the proposed, procedures require any special arrangements 
and, 1-f so, with whom? 

0 For each Intended user, what particular kinds of Information and in 
what kinds of format will be deemed relevant? 

0 What kinds of dialogue, via what techniques, will best match users' 
routine information processing styles? 

While the evaluation process 1s underway; the superintendent 1n our 
scenario, or any other organizer, would constantly monitor the process. He 
or she would ensure that the evaluation is proceeding 1n light of how 
previously raised questions were answered; determine 1f any unanticipated 
factor influence 1s beginning to emerge; determine 1f an expected influence 
is less than anticipated, and 1f resources might be safely shifted to 
another factor of concern. 
Adding the Finishing Touches 

This activity 1s the final phase in maximizing the potential for 
evaluation use. The group of factors of interest here represents that 
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point 1n the evaluation process where most, or all, of the evaluation 
Information has actually been collected. That information must now be 
communicated in such a way that the designated users will actually apply 
the information. f 'v%. 

Among the questions the administrator-organizer should consider here 

are: 

° What combination of written and oral reporting will most enhance 
use of Information? 

° At what t1me(s) should these teports be provided? 

° After the reports are provided, will any final arguments be needed 
to convince users to act on the Information? 

Finally, note that the answers arrived at 1n any one stage will 

Influence questions and organizing strategies stemming from a subsequent 

stage. Further^the process 1s cyclical and permits specifications 

proposed at an earlier stage to be modified (e.g., stressing/ de-emphasizing 

one of the evaluation questions) 1n light of subsequent planning, conduct, 

and emerging receptivity toward the evaluation and Its use. 

CONCLUSION 

We have suggested here that evaluation can serve a variety of educa- 
tional management question^ and outlined some of the question areas. We 
have stressed that an administrator's ability to use evaluation as a 
management tool depends not only upon his or her own perception of evalua- 
tion but also on the perceptions of other potential evaluation users in the 
system. Evaluation's contribution as a management tool is affected by the 
degree to which evaluation comes to be accepted and used throughout the 
various levels of the system. 
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Research has uncovered a variety of factors Influencing an evalua- 
tion's use potential. These factors reflect human considerations — such 
as people's attitudes toward and expectations for the evaluation; context 
considerations — such as an evaluation's requirements within a particular 
setting; and evaluation considerations — such as procedures used and means 
of communicating Information. The research demonstrates that an evalua- 
tion's use potential, and therefore Its application as a management tool, 
can be greatly enhanced 1f someone takes responsibility for organizing the 
evaluation to meet specified needs, for particular, users, 1n light of the 
factors operating 1n the given setting. The research also suggests that a 
program-level administrator 1s 1n the most strategic position to assume 
this responsibility. 4 

It seems evident that administrators and evaluators must come to know 
more about each other's operational needs and viewpoints. To the extent 
that administrators and evaluators share responsibility for setting an 
evaluation's fod and purposes, and ensure that the evaluation addresses 
these purposes, the evaluation's dedsl on-making power and relevance are 
Increased. 

When a program evaluation is being considered, therefore, the 
administrator needs to decide on Its various audiences, determine their 
questions and Information needs, and anticipate the uses they are likely to 
make of the Information. At the same time, the administrator needs to 
consider the factors existing 1n the given setting that are likely to 
influence these uses. The framework we suggest can be applied to organize 
the evaluation so as to have a high potential for meeting the intended uses 
in light of various factor Influences. 

u 
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